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I. INTRODUCTION 


The USL NASA PC software evaluation project is intended to 
provide a structured framework for facilitating the development 
of quality NASA PC software products. The project will assist 
NASA PC development staff to understand the characteristics -^nd 
functions of NASA PC software products. Based on the results of 
the project’s evaluations and r ecomnendat i ons , users can judge 
the reliability, usability, acceptability, maintainability and 
customizability of all the PC software products. 

The objective of this report is to provide initial, 
high-level specifications and guidelines for NASA PC software 
evaluation. The primary tasks to be addressed in this project are 
as foil ows : 

(l) To gain a strong understanding of what software evaluation 
entails and how to organize a structured software evaluation 
process. 
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(2) To define a structured methodology for conducting the 
software evaluation process. 

(3) To develop a set of PC sofware evaluation criteria and 
evaluation rating scales. 

(4) To conduct PC software evaluations in accordance with the 
identified methodology. 

This report presents a method for evaluating NASA -^PC 
software products. Chapter II introduces the categorization of 
the NASA PC software. Chapter III discusses the evaluation 
criteria to be used for NASA PC software. The Appendices to this 
report provide detailed comparison charts for each category of 
NASA PC software. The evaluation process is presented in Chapter 
IV. 
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II. NASA PC SOFTWARE CATEGORIZATION 

In this evaluation project, the NASA PC software products 
are basically divided into ten categories. They are: 

** Database/Fi 1 e Managemet Systems 
** Operating Systems 
** Programming Languages 
** Statistical Data Analysis 
** Document Processing 
** Communication Packages 
** Network System Software 
** Graphics Support Software 
** Environment Management Packages 
** Project Management Systems 
** General Utilities 
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III. EVALUATION CRITERIA FOR NASA PROJECT SOFTWARE 

3 . 1 Tniminn Evaluation Criteria for NASA Project Software 

To evaluate product within the software categories mentioned 
in Chapter II, selected measures of performance (criteria) were 
developed. Various criteria are applicable to all categories of 
NASA project software, for example: ^ 

** Ease of Use 
** User Friendliness 
** Documentation 
** Vendor Support/Update 
** Machine Compatibility 

3.1.1 Ease _oX Use 

Ease of use is one of the most important characteristics of 
any widely accepted software product. The evaluators should 
consider the following subcriteria: 

** Menu-Driven System -- Menu refers to a list of choices from 
which to chooce an action that the program will perform. A 
good menu-driven software enables the user to have easy 
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control over the way the program proceeds. 

** "Help” Facility -- Good software should have "help” 
capabilities to assist users to understand the functions of 
various conmands. 

** On-line Tutorial -- This facility teaches users how to use 
the software. 

** Keyboard Functions -- With this facility, users can perform 
various functions by simply pressing the corresponding keys. 

3.1.2 User Friendliness 3 

There are many subcriteria to be considered for evaluating the 

user friendliness of software: 

** "Help” Facility -- This capability assists users to know how 
to use the software system. 

** Interactive Dialogue -- This capability interacts with the 
user via questions, warning, comnents, etc. 

** Error Messages -- This capability displays error messages if 
errorous conmands are entered. 

** Protection -- A good program will not allow the user to 
delete or modify data accidentally. 

3.1.3 Document at i on 
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Documentation is very important to enable productive use of 

software products. The following subcriteria should be 

considered [DataPro, 83]: 

** Index -- all important terms should be listed in an index. 

** Illustrations -- Drawings or photographs should be provided 
to help the user to understand the concepts presented. 

* * Ex amp les -- it is necessary to provide ex amp les to explain 
certain functions. 

** Glossary of Terms -- These should be provided as a reference 
source. ^ 

** Quick Reference Guide -- This can be any size of pocket card 
containing enough information to enable the user to use the 
software without referring to the full reference manual. 

3.1.4 V .mdor Su pport/Update 

Subcriteria include [DataPro, 83]: 

** Updates of Product -- Vendors should provide free upgrades 
and product enhancement when they are avaliable. 

** On-line Tutorials -- These are usually provided on separate 
diskette to help users understand and use the program. 

** Newsletters -- Vendors should offer newsletters to help the 
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user to use the product more efficiently. 

3.1.5 Machine Compatabi 1 ily 

There are four types of IBM microcomputer configurations in 
the USL NASA PC R&D environment. Evaluators must check whether 
the software product being evaluated is compatible with the 
following machines: 


* * 

1HV1 

PC 

* * 

IBM 

XT 

* * 

IHV1 

AT 

* * 

IBM 

3270/G 
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There are certain specific criteria that are dictated by 

individual application. For example, security may be a basic and 

fundamental criterion in a database management system, but word 

processing may be irrelevant in such an environment. Therefore, 

different software products categories should be evaluated 

seperately [DataPro, 85]. The sets of criteria for each 

1 

software product category are listed in the comparison charts ~»in 
the appendices to this report. 


In order to have an objective judgement, a standardized and 
quantitative evaluation scale should be defined. A criterion was 
judged to be quantitative if, for example, a definite answer of 
"yes” or "no” could be given without subjective judgement or if a 
numeric value could be objectively derived [Edmonds, et. al., 
84] . 
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IV. EVALUATION PROCESS 


The NASA PC software evaluation process is as follows: 

(1) Understand the characteristics and functions of the 

software product to be evaluated. This information can be 
obtained from the following sources: 

** Documents and manuals from vendors ^ 

** Evaluation reports, such as DataPro Research Reports and 
Auerbach Dataworld Series. 

** Interactive testing -- Interactive testing of the 

software product. 

** Survey of other users’ experiences. 

(2) Evaluate the software product according to each criterion in 
the comparison chart. Quantitative evaluation results should 
be entered into the chart. 

(3) Highlight the evaluation result of each entry in the 
comparison chart, if necessary, by providing information in 
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the remarks columns. 

(4) Weight and rate the software product. For each product to be 
evaluated, first assign a set of weight percentages to each 
corresponding criterion to reflect its relative importance. 
Then assign a score for each criterion which reflects how 
well the product satisfies the criterion. 

Each criterion may be further divided into criteria of a 
lower level, which can still be further divided. In s-^ch 
cases, the weight percentages and scores for the lowest 
criteria categories are entered and computed first. By 
totaling the weighted scores for the lower level criteria 
within the same category, the score for the corresponding 
higher level criterion can be obtained. The weighted score 
for each higher level criterion can then be obtained by 
multiplying its weight percentage by its score. The process 
is repeated until the final (highest level) total weighted 
score is obtained [DataPro, 85; Edmonds, et. al., 84]. 

(5) Write the final evaluation report. The evaluators sunmarize 
each evaluation product after the above steps are completed. 
The final report should be an analytical discussion of the 
product and how it compares to comparable products in the 
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NASA PC R&D software environment. 


Based on the above evaluation process, a very detailed 
analysis and productive r econmendat i on can be generated. This 
will assist NASA PC development staff to understand the 
chara t e r i s t i c s and functions of supportive NASA PC software 
products. This will also provide users with a basis to judge the 
reliability, usability, acceptability and ma inta inbi 1 i ty of the 
evaluated NASA PC software products. ~f 
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APPENDICES: 


The Comparison Charts that perform the side-by-side comparison 
of all NASA PC software within a particular software category are 
presented in the appendices to this report. A set of evaluation 
criteria are listed on one axis, and the names of the evaluated 
software products are listed on the other axis. Evaluators should 
quantitatively assign the result of the evaluation to e-^ch 
criterion. 
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Product Name 


Manufacturer 


| Informix - SQL| R:base 5000 | Oracle 


(Relational \ | 

(Database Mana-| | 

(gement System | MicroRim Inc.|Oracle Corporation 


(System Characteristics 


( Mininum Memory Required 


| Memory Required Program 
| after Loading 


(Machine Compatibility 




IBM 3270/G 


(Program Characteristics 


| Multi-User 


Type of Database 
Structure 


Access Language 


Automatic Update and 
Integrity 


(File Characteristics 


No. of Fields per Record 


No. of Char, per Fields ) 


| No. of Records per File 


| No. of Files Concurrently! 
| Opened | 
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No. of Fields per Index 
Key 

No. of Characters per 
Index Key 

No. of Index Keys per 
File 

File Structure 
Modification 

Add Field Values 
Add Records 
Delete Field Values 
Delete Records 
Modify Field Values 
Modify Records 
Types of Fields 
Nuneric 
Character 
Sorting/Indexing 
Ascending 
Descending 

Based on a Single Field 

Indexing Based on 
Multiple Fields 

| File Ccupatibi 1 ity 

| ASCII 



| DIF 
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| DBF (dBase II) 

| Secur 1 ty 
| User Passwords 
| Levels of Security 
| Write Protection 
| Data Encryption 
IDocunentation 
| Index 

| Illustrations 
| Exanples 
Glossary 
Pocket Guide 
| Full Reference Manual 
| Vendor Support 
| Updates 

On-Line Tutorials 
Newsletters 
(User Friendliness 
| Interactive Dialogue 
Error Messages 
| Protection 

"Help" Facilities 
| Ease of Use 
| Menu-Driven System 
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On-line Tutorials 


"Help" Facilities 


Keyboard Functions 


Data Installation 


First Installation Date 


Nunber Installed to Date 


NAME OF EVALUATOR 


DATE 


1 
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APPENDIX B. OPERATING SYSTEMS COMPARISON CHARTS 
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APPENDIX C. PROGRAMMING LANGUAGES COMPARISON CHARTS 


■i 
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APPENDIX D. STATISTICAL DATA ANALYSIS COMPARISON CHARTS 
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APPENDIX E. DOCUMENT PROCESSING CCMPAR I SON CHARTS 
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APPENDIX F. CCMvRJN I CAT I ON PACKAGES COMPARISON CHARTS 
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APPENDIX G. NETWORK SYSTEM SOFTWARE COMPARISON CHARTS 
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APPENDIX H. GRAPHICS SUPPORT SOFTWARE COMPARISON CHARTS 
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APPENDIX I. ENVIRONMENT MANAGEMENT COMPARISON CHARTS 
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APPENDIX J 


GENERAL UTILITIES COMPARISON CHART 
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